Lifestream annotation method and system

ABSTRACT

A method and system for collecting information, enriching the collected information, and linking the enriched information is provided. The system includes an application server that is accessible by users via their computer device for performing a variety of functions encompassed in a lifestream data function. Lifestreams (or lifestream data) contains notes, recordings and annotation data elements. The goal of the lifestream data functions is to ultimately enrich the individual data elements in each lifestream which then form a network of enriched lifestreams or aspects of enriched lifestreams to provide a user with: (a) a better organization of his/her data, notes, thoughts and intentions, (b) a better search or filter tool to search for data, notes, thoughts and intentions, and/or (c) a better ranking mechanism for ranking or determining relevance of data, notes, thoughts and intentions.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.13/345,023, filed Jan. 6, 2012, which claims priority from U.S.Provisional Patent Application 61/460,813, filed Jan. 7, 2011, which isincorporated herein by reference. U.S. patent application Ser. No.13/345,023 is a continuation-in-part of U.S. patent application Ser. No.12/322,046 filed Jan. 27, 2009, which is incorporated herein byreference.

FIELD OF THE INVENTION

The invention relates generally to online social media. Moreparticularly, the present invention relates to collecting and enrichinginformation.

BACKGROUND

Today, a tremendous number of online services and applications areavailable. The existing online services are related to virtually everyaspect of life, including work, relationships, health, entertainment,news, etc. Access to these online services typically only requires anInternet connection. However, the abundance of services also causesinformation overload. As a result, disorganized information is floatingaround many users' lives and brains. Instead of positively influencingour lives, the services become cumbersome and are often neglected. Inparticular, the advent of social media has made it easier to shareinformation with more people and make connections with people who weotherwise would not. However, the ubiquity of social media (such asthrough social networks) has created a further explosion in theinformation content of our lives and relationships. This informationclutter reduces a user's productivity and shortens the user's attentionspan.

Typically, ideas begin with private thoughts that are later refined andshared to friends and, perhaps, the public at large. Who these ideas areto be shared with often evolve over time. Existing online services arenot amenable to this process of developing ideas and deciding where toshare them at a later time. For example, existing social media forcesideas immediately into the public domain. Information or media posted onsocial networks (e.g. Facebook.com, Twitter.com, and Flickr.com) orblogs (e.g. Wordpress.com and Blogger.com) are immediately accessible byother users. To post information in existing social media requiresupfront cognitive decision-making related to the audience level of theinformation at the time the information was collected or generated. Thisprocess does not match the way human brains function.

Systems, such as emailing to oneself and note-taking applications e.g.Evernote.com) exist for users to store private thoughts. However, theseexisting systems are not integrated with other services or applications.In other words, a user would have to access and sign on to multipleapplications simultaneously to utilize information stored in theapplication storing the private thoughts. Because of the lack ofintegration and inconvenience of these systems, they are seldom used,therefore, ideas are often forgotten. In addition, existing note-takingapplications require users to provide all of the information content tothe note. By forcing a user to provide all of the information content,the note-taking process is typically tedious, error-prone, andincomplete.

The present invention addresses at least the above-described difficultproblems and advances the art with a semantic note taking system andmethod.

SUMMARY OF THE INVENTION

The present invention is directed to a semantic note taking system andmethod for collecting information, enriching the information, andbinding the information to services. A plurality of users arecommunicatively connected to an application server to create one or morenotes that can be bound to one or more of a plurality of services andstored in a database. The application server operates a plurality offunctions, including a note taking function for allowing each of theusers to create one or more notes, a categorizing function for labelingeach of the notes with one or more changeable categories, a contextfunction for associating one or more context traits with each of thenotes, a binding function to establish one or more changeable bindingrules for each of the notes, wherein the binding rules determine one ormore of the services where the note is to be bound, and wherein thebinding rules are related to the content of the note, the categories ofthe note, the context traits of the note, a user binding selection, orany combination thereof, and a communication function for communicatingeach of the notes to one or more of the services where the note is to bebound.

A note includes text, audio media, visual media, audio-visual media,recorded data, a weblink, a pointer to an information source, or anycombination thereof. In an embodiment, one or more of the context traitsassociated with the note includes a time, a location, physical datarelating to the user device used to generate the note, or anycombination thereof. In another embodiment, relevant data is derivedbased on one or more of the context traits and the relevant data isattached to the note. The relevant data can be derived by matching oneor more of the context traits with data from an information module suchas a calendar, an address book, a contact list, a user profile, a userhistory, or any combination thereof. In an embodiment, one or more ofthe context traits associated with the note is based on one or morecategories of the note, the content of the note, a user history, or anycombination thereof.

In a preferred embodiment, the binding rules include one or morepublication properties, such as an audience level for the note and/orone or more locations to publish the note. The locations to publish thenote can include a private domain of the user, one or more friends ofthe user, one or more computer-implemented social networks, a blog, anonline discussion board, a website, or any combination thereof. In anembodiment, the binding rules are automatically determined based atleast partially on one or more of the categories, one or more contexttraits, the note content, a user history, or any combination thereof.

The application server of an embodiment of the present inventionoperates a suggestion function for suggesting one or more services,binding rules, and/or categories. The suggestions can be based at leastpartially on one or more the categories, one or more of the contexttraits, the note content, a user history, one or more binding rules, orany combination thereof. In an embodiment, a note is automatically boundto one of the suggested services. Preferably, the suggestion functionuses a user history of selections of services and/or categories.

In an embodiment of the present invention, a user interface is providedto allow user entry of the categories and binding rules for a note,wherein the user interface includes a first symbol associated with thecategories of the note and a second symbol associated with the bindingrules of the note. Preferably, the user entry of the first symboltriggers a display of one or more of the categories suggested to theuser and selectable by the user. Similarly, the user entry of the secondsymbol triggers a display of one or more of the binding rules suggestedto the user and selectable by the user. In another embodiment, a userinterface corresponding to each of the services where a note can bebound is provided. Each of the service-specific user interfaces includesone or more buttons for initiating an action for the servicecorresponding to the user interface. Preferably, a single click of oneof the buttons initiates creation of the note, automatic addition ofcontent to the note, automatic labeling of the note with categories, andcommunication of the note to the service corresponding to the userinterface.

In an embodiment, the user device operated by users to access theapplication server is a mobile device that communicates with theapplication server over a wireless network. The services where a notecan be bound are selected from the group consisting of a website, a webapplication, a computer-implemented social network, a blog, a reviewwebsite, a product review website, an entertainment website, a healthapplication, a medical application, an online retailer, an emailapplication, a research application, a clinical application, a calendar,and an address book.

In an alternate embodiment a method and system is provided forcollecting information, enriching the collected information and linkingthe enriched information. The system includes an application server thatis accessible by a plurality of users each with their own computerdevice for performing a variety of functions encompassed in a so-calledlifestream data function. Lifestreams (or lifestream data) containsnotes N₁, N₂ . . . N_(N), recordings R₁, R₂ . . . R_(M) or segments ofrecordings R_(S1), R_(S2) . . . R_(SM) and annotation data elements A₁,A₂ . . . A_(P), or any combination thereof. The goal of the lifestreamdata functions is to ultimately enrich the individual data elements ineach lifestream which then form a network of enriched lifestreams oraspects of enriched lifestreams to provide a user with: (a) a betterorganization of his/her data, notes, thoughts and intentions, (b) abetter search or filter tool to search for data, notes, thoughts andintentions, and/or (c) a better ranking mechanism for ranking ordetermining relevance of data, notes, thoughts and intentions.

BRIEF DESCRIPTION OF THE FIGURES

The present invention together with its objectives and advantages willbe understood by reading the following description in conjunction withthe drawings, in which:

FIG. 1 shows an example of a semantic note taking system according tothe present invention.

FIG. 2 shows an example of mutable binding of notes to multiple servicesaccording to the present invention.

FIG. 3 shows a flow chart of an example note creation, enrichment, andbinding according to the present invention.

FIG. 4 shows a flow chart of example steps taken by a user according tothe present invention.

FIG. 5 shows an example of a suggestion function according to thepresent invention.

FIG. 6 shows an example user interface for semantic note takingaccording to the present invention.

FIGS. 7A-B show examples of semantic skin user interfaces correspondingto services according to the present invention.

FIG. 8 shows the core services of an exemplary embodiment of the presentinvention.

FIG. 9 shows the technology services underlying the core services shownin FIG. 8.

FIG. 10 shows a flow chart of an example login and authenticationfunction in an exemplary embodiment of the present invention.

FIG. 11 shows a flow chart of an example Queue service in an exemplaryembodiment of the present invention.

FIGS. 12A-B show an example of using semantic note taking with onlineretailers according to the present invention.

FIG. 13 shows an example of a system for collecting, enriching andlinking information according to the present invention.

FIG. 14 shows an example of a plurality of notes N₁, N₂ . . . N_(n),recordings R₁, R₁ . . . R_(m), and annotation data elements A₁, A₂ . . .A_(P) according to the present invention.

FIG. 15 shows an example of a linked network of enriched lifestreams ofnotes N₁, N₂, N₃, segments derived from recordings Rs₁, Rs₂, Rs₃, andannotation data elements A₁, A₂, A₃, according to the present invention

FIG. 16 shows an example of a recording 1600 in which segments 1610,1620 are derived according to the present invention.

FIG. 17 shows an example of a data capture device 1700 according to thepresent invention, which is merely one example of the variety ofpossibilities. In this example 1710 pertains to sensing speaker andrelative speaker position, 1720 pertains to measuring respirationpatterns, 1730 pertains to measuring pupil dilation, blinking, tears,redness or the like, 1740 pertains to audio/video stream recording, 1750pertains to location, direction, rotation, tilt, vibration or the like,1760 pertains to smartphone transmission, and 1770 pertains toelectrical stress and arousal measurements.

DETAILED DESCRIPTION OF THE INVENTION

The abundance of online services provides users with opportunities toperform many tasks and connect with numerous people from nearly anylocation. However, the large number of services often cause informationoverload and a user cannot efficiently organize the explosion ofinformation content concerning the user's life, work, and relationships.The present invention is directed to a semantic note taking system andmethod to collect, enrich, and bind information to services. Semanticnote taking allows users to effortless offload information from privatethought streams for later determination of where to bind these thoughts.In addition, semantic note taking minimizes the friction to record andshare ideas by automatically enriching the note and reducing unnecessaryuser input. In other words, the present invention is directed to asystem and method to collect unscripted data, add more meaning and useout of the data, and bind the data to services.

FIG. 1 shows a preferred embodiment of a semantic note taking system 100according to the present invention. The semantic note taking system 100includes an application server 110 that is accessible by a plurality ofusers U₁-U_(N) for creating one or more notes, categorizing the notes,and binding the notes to one or more services S₁-S_(M). The usersU₁-U_(N) connect to the application server 110 through any devicecapable of being communicatively connected to a network, such as theInternet. The user devices can include a computer, a laptop, a personaldigital assistant (PDA), a cell phone, a mobile device, or a smartphone. FIG. 1 shows user U₁ with a mobile phone 140 that is connected tothe application server 110, such as through a wireless network.

A note taking function, operated by the application server 110, allowseach of the users U₁-U_(N) to create one or more notes. Notes caninclude text, audio media, visual media, audio-visual media, recordeddata, a weblink, a pointer to an information source (e.g. a bookmark),or any other type of data. Notes created by the users U₁-U_(N) aretransmitted from the user devices to the application server 110 andstored in the database 420 for later binding to one or more of theservices S₁-S_(M).

The application server 110 also operates a categorizing function tolabel the notes with one or more categories. In an embodiment, thecategorizing function allows users U₁-U_(N) to label the notes, thoughautomatic or suggested note labeling is also possible. It is noted thatone or more of the categories of each note is changeable. Labeling anote with categories enriches the note. The note is further enriched bythe context function operated by the application server 110, whichassociates one or more context traits with the note. Context traitsassociated with a note can be based on one or more of the categories ofthe note, the content of the note, a user history, or any combinationthereof. Examples of context traits include a time, a location, andphysical data relating to the user device. For example, the mobile phone140 of user U₁ is capable of measuring the location where the note wascreated (e.g. by using GPS measurements) and this location can beautomatically associated with the note without user input, therebyenriching the note.

Context information is not restricted to a time-stamp or location-stampassociated with the note. In a preferred embodiment of the presentinvention, relevant data is derived based on one or more of the contexttraits. Relevant data can be derived by matching one or more of thecontext traits with data from an information module 130, such as acalendar, an address book, a contact list, a user profile, a userhistory, or any combination thereof. For example, a time context traitwhen the note is created is matched with a meeting time indicated by acalendar information module. Because the calendar includes relevantdata, e.g. the participants of the meeting and the discussion topics ofthe meeting, this relevant data can be attached to the note to furtherenrich the note.

A binding function is provided to establish one or more binding rulesfor a note. The binding rules determine one or more services S₁-S_(M) orapplications where the note is to be bound. The binding rules arerelated to the content of the note, the categories of the note, thecontext traits of the note, a user binding selection, or any combinationthereof. In an embodiment, the binding rules of a note are automaticallydetermined based at least partially on one or more categories of thenote, one or more context traits, the content of the note, the userhistory, or any combination thereof.

The services S₁-S_(M) where a note can be bound can include a website, aweb application, a computer-implemented social network, a blog, a reviewwebsite, a product review website, an entertainment website, a healthapplication, a medical application, an online retailer, an emailapplication, a research application, a clinical application, a calendar,an address book, or any combination thereof. A note that is bound to aservice is communicated to the service. What is done with the note afterit is communicated depends on the particular functions of the serviceand the content of the note. For example, a note bound to a blog couldbe published on the blog and a note bound to a research application caninclude data for analysis by the research application. In addition tocommunicating a note to a service, in an embodiment of the presentinvention, data can be extracted from the service where the note isbound and the extracted data is added to the note.

The services S₁-S_(M) shown in FIG. 1 are generally not integrated andsome of the services S₁-S_(M) where a note can be bound may also requireuser login or authentication. To facilitate the integration of theservices and allow access to the services, in an embodiment, theapplication server 110 includes a login function to provide logininformation for access to the services. Integration of the servicesenables a user to collect or create information content and offload thecontent to a single location, instead of having to interact with amyriad of different services separately.

In a preferred embodiment of the present invention, the binding rulescomprise one or more publication properties. The publication propertiesinclude an audience or privacy level for the note and the location topublish the note. FIG. 2 shows three notes N₁, N₂, and N₃ created byuser U₁ and the binding of these notes to different services. Theaudience levels generally include private, social, and publicdesignations. The locations to publish the note includes a privatedomain of the user, one or more friends F₁-F₃ of the user, one or morecomputer-implemented social networks, a blog, an online discussionboard, a website, or any combination thereof. FIG. 2 shows note N₁ bound210 to a private journal and bound 220 to an email application. In otherwords, the binding rules for note N₁ indicate that the note should becommunicated to the journal service and the email application with theappropriate audience levels.

A note can be bound to any number of services at any time. Importantly,the binding of the note is changeable. In other words, a user can changethe binding rules to alter the location where a note is to be bound. InFIG. 2, note N₁ is originally bound 210 to a journal but is later bound230 to Social Network 1. Similarly, note N₃ is originally bound 240 to ablog, but is later bound 250 to an electronic commerce website. Themutable binding rules and late-binding method of the present inventionis in accord with how ideas are typically generated in our brains. Inparticular, ideas and information typically start as private thoughtsthat are later shared and discussed with people we know. Finally, theinformation is published to the world. This is in contrast to existingsocial media where generated media are forced immediately into thepublic realm.

FIG. 2 also shows note N₁ published and communicated to associates orfriends F₁-F₃ of user U₁. By publishing note N₁ to friends F₁-F₃, an adhoc community or an ad hoc social network is developed around note N₁.In an embodiment, one or more of the friends F₁-F₃ can change the noteN₁. In particular, data can be extracted from one of the friends F₁-F₃and the extracted data is added to the note N₁.

FIGS. 3 and 4 show flow charts for exemplary methods of creating,enriching, and binding the notes according to the present invention. Thesteps circled with a dashed line 310 in FIG. 3 show the enrichment ofthe notes by labeling the note with categories, associating contexttraits with the note, and deriving relevant data based on the contexttraits. FIG. 4 shows the steps undertaken by a user of the semantic notetaking system of the present invention.

FIGS. 3 and 4 also include the steps of suggesting binding rules andreceiving suggestions, respectively. The application server operates asuggestion function for suggesting one or more services where the notecan be bound to the user who created the note. FIG. 5 shows the inputsfor the suggestion function 510, which can include the content of thenote, the categories of the note, the context traits, and/or the userhistory. Based on these inputs, the suggestion function 510 suggestsbinding rules 520, such as a list of suggested services where the notecan be bound. For example, FIG. 5 shows a ranked list of services with“Service 5” being the top-ranked service where the note can be bound.

In an embodiment, the suggested services or binding rules are selectableby the user, where the user selection, binds the note to the selectedservice. User selections can be stored for reference by the suggestionfunction. For example, a first note is created by a user, labeled with“Category 4”, and bound to “Service 5” by user selection. Because ofthis user selection, “Service 5” would be suggested for a second notealso labeled with “Category 4”. It is noted that the first and secondnotes can be created by the same user, or they can be created bydifferent users. In other words, the suggestion function 510 can rely onthe user history of the same user or the user history of a community ofusers. It is noted that the association of a service to a category ismutable and evolvable depending on user behavior. For example, though“Category 4” and “Service 5” are associated in the above example, theassociation of services for “Category 4” could change to “Service 3” ifusers frequently alter the selection from “Service 5” to “Service 3” fornotes having a label of “Category 4”.

In an embodiment, a note is automatically bound to one of the suggestedservices without user selection. In a preferred embodiment, the storageof past user selections enables intelligent and automatic binding ofnotes to services. Automatic binding of notes to services reduces theneed for user input.

The suggestion function 510 is also capable of suggesting categories tolabel a note in addition to or replacement of suggesting binding rules.Suggestions of categories can be analogous to the suggestions ofservices described in the previous paragraphs. In particular,suggestions of categories can be based on the content of the note, thecontext traits of the note, other categories of the note, a userhistory, the binding rules established for the note, the servicesdetermined by the binding rules, or any combination thereof. Forexample, a note contains a description of a product. Based on this notecontent, a category named “product” is suggested to the user forlabeling the note. In another example, the association of a service anda category based on a user history can be used to suggest one or morecategories for notes having binding rules with that particular service.In an embodiment, a note is automatically labeled with one or moresuggested categories.

FIG. 6 shows an example user interface 600 for user creation of a noteand user entry of categories and binding rules for the note. In anembodiment, the user interface 600 is accessible via a web browser.However, alternative methods of accessing user interfaces known in theart may also be used. In a preferred embodiment, the user interface 600allows user entry of a first symbol 610 to designate entry of a categoryand a second symbol 620 to designate entry of a binding rule. As shownin FIG. 6, the first symbol 610 is the “#” sign and categorization isaccomplished by placing a “#” symbol adjacent to a word. Similarly, thesecond symbol 620 is the “@” sign to mark a service for a binding rule.The first 610 and second 620 symbols allow easy entry of categories andbinding rules. As can be appreciated by one or ordinary skill in theart, the present invention is not limited to the use of symbols forcategorization and binding rules.

In a preferred embodiment, entry of the first symbol 610 triggers a listof previously used categories to be displayed to the user. Inparticular, the list can be filtered to only include categories that maybe relevant to the note, such as based on the content of the note or thecontext traits of the note. In an embodiment, users can overridesuggestions and label the note with non-suggested category or a newcategory. Similarly, entry of the second symbol 620 triggers a list ofservices or binding rules to be displayed to the user. In a preferredembodiment, user selection of a category to label the note triggers adisplay of suggested services where the note can be bound. Conversely,user selection of a binding rule triggers a display of suggestedcategories to label the note. The suggestions can be based on previouslystored user selections, i.e. a user history, or any other bases asdescribed herein with respect to the suggestion function.

The user interface 600 also includes a list of categories 630 used inall stored notes. A user can select a category from the list 630 to viewnotes labeled with the selected category. In an embodiment, a search box640 is also available to find previously saved notes. FIG. 6 also showspreviously saved notes 660 and 680. The note 660 includes a photo 665and a weblink 670. The note 680 includes text description of a wine andis labeled 690 with the categories “wine” and “product” by use of thefirst symbol. The user interface 600 also includes a user name 650, suchas the user's email address. In an embodiment, the application serverrequires a user authentication to access the semantic note takingsystem.

Embodiments of the present invention are also directed to userinterfaces corresponding to services to facilitate simple collection ofdata and binding to services. In an embodiment, a user interface isprovided for each of the services where a note can be bound. The userinterface corresponding to a particular service includes one or morebuttons for initiating an action specific to that service. Servicespecific user interfaces are referred to as semantic skins. In apreferred embodiment, a single click of a button on a semantic skininitiates the creation of a note, automatic content generation andaddition to the note, automatic labeling of the note with one or morecategories appropriate to the service, and communication of the note tothe service.

FIG. 7A shows an example of a semantic skin 710 for a social networkservice and FIG. 7B shows an example of a semantic skin 750 for a healthor diet application. The semantic skin 710 includes a button 720 to“poke” a friend of the user in the social network, a button 730 tosubmit the current status of the user, and a button 740 to post a phototo the social network. The semantic skin 750 in the example shown byFIG. 7B includes a button 760 to enter exercise activity, a button 770to enter diet information, and a button 780 to measure medical readings.The semantic skins can rely on context traits derived from the devicewhere the data entry is performed. For example, the button 730 ofsemantic skin 710 can utilize the current location of the user asmeasured by the user device and publish that location to the so network.In another example, a user is operating a device that includes a heartmonitor and the button 780 extracts heart rate measurements, attachesthe measurement to a note, and communicates the note to a healthservice.

In a preferred embodiment, buttons in user interfaces and semantic skinsare programmable. The buttons serve the purpose of providing simple orsingle click entry of data and minimizes the friction for a user tocollect information and bind the information to services.

FIGS. 8-11 include details of a functional embodiment of the presentinvention. The exemplary embodiment shown in FIGS. 8-11 are directed tosocial network services, such as Facebook.com and Twitter.com, thoughits extension to other services can be appreciated by one of ordinaryskill in the art. The present invention is not limited to the embodimentof FIGS. 8-11; alternatives to the services and applications describedin FIGS. 8-11 can be identified by one of ordinary skill in the art.

FIGS. 8-9 show the technologies used by the application server in anexemplary embodiment of the present invention. The core services areshown by FIG. 8 and include the AuthenticationService, NodeService,NodeMetaService, LabelService, SearchService, UndoManager, QueueService,LogService, and PrivacyService. Each of the core services has its owninstance of a BerkeleyDB Java Edition as a backing store. FIG. 9 showsan initial user request 910 that is handled by a web server, e.g.lighttpd. Lighttpd serves up all the static content. If the request isfor dynamic content then it is forwarded to the jetty Java ApplicationServer. Jetty runs inside of the Java Virtual Machine. In the embodimentshown in FIG. 9, the struts2 web framework is used inside of Jetty.Struts2 determines which services are needed to fulfill the request anduses Google Guice to create all the objects needed by the user. Afterall the objects have been created, struts2 fulfills the users requestand generates the page. The jQuery JavaScript library is used to speedup and simplify cross-platform development. Google collections Libraryis used to augment the standard Java collections.

FIG. 10 shows a flow chart related to the AuthenticationService of anexemplary embodiment of the present invention. The AuthenticationServiceallows a user to communicate with any service that requiresauthentication or login information; thereby the semantic note takingsystem can be integrated with any number of third party applications andservices. In the example shown in FIG. 10, a user request to share anote on Facebook is shown. The request is handled by struts2, whichchecks if the user is a valid user of the semantic note taking system.If the request is from a valid user, a request is sent to theFacebookService to verify if the user has valid Facebook credentials. Ifthe user has a valid Facebook account, the request is sent to theQueueService to en-queue the Facebook share request.

FIG. 11 shows a flow chart related to an embodiment of the QueueService.When an event is en-queued in the QueueService's queue, the QueueServicechecks if the note associated with that event has been previously sharedto Facebook. The LogService tracks when a note has been successfullyshared to Facebook. If the note has been previously successfully shared,it is removed from the queue. The QueueService then checks if there wasa problem sharing the note. If the note has had a failure, the shareevent is timed out and put hack in the queue to be run after X minutes.The note is then pushed to the user's Facebook account. The note ispushed by the QueueService calling the FacebookService to fulfill theshare request. If the request succeeds, the event is removed from theQueueService's queue and logged to LogService.

EXAMPLES

The following examples demonstrate exemplary applications for theembodiments of the present invention. It is noted that applications forthe present invention are not limited by these examples; otherapplications of the present invention can be identified by one ofordinary skill in the art.

Example 1: Product Review

A user enjoys a product, such as a glass of wine. During a dinnerengagement the user jots down information related to the wine, such asthe brand, vintage, variety, etc., by accessing a user interface to thesemantic note taking system. The note includes a label of “#wine” forfuture reference to the note. Context traits, including the GPScoordinates where the note was created is associated with the note.Using the GPS coordinates and accessing a directory information module,the name of the restaurant is derived and automatically associated withthe note. The note is saved in the database.

At a later date, the user chooses to discuss the wine with friends ofthe user who may be wine enthusiasts. By binding the wine review note toa social network or an email application, the note is communicated toone or more friends of the user. One of the friends writes a briefreview of the wine and attaches that review to the note. After the notehas been shared to friends of the user, the user decides to write a winereview to be posted on a wine website. The user refines the note andbinds it to the wine review website to be published to the world.

Example 2: Retail

FIGS. 12A-B show an example of a retail application with bidding frommultiple online retailers 1230. In FIG. 12A, user U₁ creates a note 1220on a user device 1210. The note 1220 includes an identification of aproduct and binding rules that indicate it is to be bound to retailersR₁, R₂, and R_(N). The note 1220 is communicated to the retailers R₁,R₂, and R_(N) based on the binding rules. As shown in FIG. 12B, afterthe note 1220 is received by the retailers R₁, R₂, and R_(N), theretailers send offers 1240 to user U₁ for the product described in thenote 1220. Alternatively or additionally, the retailers R₁, R₂, andR_(N) can send advertisements, coupons, or any information relating tothe product to the user U₁.

Example 3: Health

The present invention can be directed to a dietary, health, or medicalservice. For example, with minimal friction, a user can submit his orher dietary behavior to be tracked by a diet service. In anotherexample, medical information can be collected at a hospital visit ordoctor's appointment and submitted to an electronic medical record (EMR)application.

Example 4: Research

The semantic note taking system of the present invention can also beapplied to a research study. In an embodiment of the present invention,research data is collected in a note. The research data can be collectedfrom the field via a mobile device. The research data is enriched withtime and location context traits. For repetitive data collection, asemantic skin would provide simple one click user entry of data. Forexample, the present invention can facilitate frictionless datagathering from door-to-door surveys. A door-to-door surveyor would neednot enter the address of each participant of the survey as locationcontext traits would be automatically associated with the note. Relevantdata (e.g. resident names and demographics) can be automaticallyaccessed based on the location context traits.

Lifestreams

FIGS. 13-17 show an alternate embodiment which can be used inconjunction with one or more or the other teachings described herein orby itself as a method or system. FIG. 13 shows a preferred embodiment ofa system 1300 for collecting information, enriching the collectedinformation and linking the enriched information. System 1300 includesan application server 1320 that is accessible by a plurality of usersU₁-U_(N) each with their own computer device 1310 for performing avariety of functions encompassed in a so-called lifestream data function(similar as described with respect to FIG. 1). In one embodimentlifestreams (or lifestream data) contains notes N₁, N₂ . . . N_(N),recordings R₁, R₂ . . . R_(M) or segments of recordings R_(S1), R_(S2) .. . R_(SM) and annotation data elements A₁, A₂ . . . A_(P), or anycombination thereof (FIG. 14). More details about each of these elementsis provided infra. The goal of the lifestream data functions is toultimately enrich the individual data elements in each lifestream, whichthen form a network of enriched lifestreams or aspects of enrichedlifestreams to provide a user with (a) a better organization of his/herdata, notes, thoughts and intentions, (b) a better search or filter toolto search for data, notes, thoughts and intentions, and/or (c) a betterranking mechanism for ranking or determining relevance of data, notes,thoughts and intentions.

Examples of lifestream data functions, which can be embodied in anycombination, are:

-   -   (i) a note taking function to allow the user to create a        plurality of notes on a computer device. Notes can be text,        audio media, visual media, audio-visual media, recorded data, a        weblink, a pointer to an information source, or any combination        thereof.    -   (ii) a recording function for collecting, recording or storing a        plurality of recordings on a computer device. Examples of        recordings are, in any combination, audio data recordings, video        data recordings, or audio-video data recordings.    -   (iii) an annotation function for collecting, recording or        storing a plurality of annotation data elements on a computer        device. Examples of annotation data element's are externally        sensed data of the environment of a user or physiological sensed        data of a user. In both cases, the sensed data is either        captured by the user or from the user using a data capture        device (e.g. FIG. 17).    -   (iv) an enriching function for enriching each of the lifestreams        and therewith creating a linked network 1500 of enriched        lifestreams (FIG. 15), which are then available to the user on        his/her computer device. Linking can be accomplished by, in any        combination:        -   (j) linking one or more segments in each of the recordings            with one or more annotation data elements, one or more            notes, or a combination thereof,        -   (jj) linking each of the notes with one or more annotation            data elements, one or more segments of the recordings, or a            combination thereof, or        -   (jjj) linking each of the annotation data elements with one            or ore notes, one or more segments of the recordings, or a            combination thereof.    -   Linking can be established by identifying commonalities in the        data elements or can be guided by a set of preferences or user        defined parameters.    -   (v) a filter or search function for filtering or searching the        enriched lifestreams or aspects of the enriched lifestreams on        the computer device.    -   (vi) a relevance function for determining relevance of the        enriched lifestreams or aspects of the enriched lifestreams in        the network of enriched lifestreams.    -   (vii) a ranking function for ranking of the enriched lifestreams        or aspects of the enriched lifestreams in the network of        enriched lifestreams.

According to a method of collecting information, enriching theinformation and linking the enriched information, a user of a computerdevice could be provided with a lifestream data function (encompassingsome or all of the above functions in any combination) to allow a userto collect a plurality of lifestreams, enrich them and obtain a networkof lifestreams or aspect of the lifestreams. Enriching can be handledeither on the computer device, by a computer (application) server or anonline service each of which is communicatively coupled with thecomputer device.

One or more segments in each recordings (e.g. 1600 in FIG. 16) can beidentified by a start and finish generating for example segment 1610 and1620. A segment in a recording can be automatically derived by afunction running on a computer device, or by a computer or applicationserver or by an online service. Specifically a segment can be derivedusing context sensing or sensing context changes in a recording, whichthen annotates the derived segment with its corresponding contextuallysensed data. Examples of context sensing or sensing context changes issensing information or changes related people, faces, location,direction, keywords, logical content blocks, or a combination thereof.

Externally sensed annotation data elements pertains for example todirection, location, identity of person or place the user is looking at.Facial recognition software, voice recognition software, GPS trackingsoftware or devices, semantic analysis software, software for analysisof timing of events or speech, directional or motion analysis software,stress indicators or voice tone analysis software, intentionalindicators or software (start, stop, etc) or the like could be used forcontext sensing or changes in context. One or more externally sensedannotation data elements could be automatically derived by the computerdevice, the computer or application server or an online service based oncontext sensing or sensing changes in context in the externally senseddata.

Physiological sensed data of the user can be obtained usingphysiological sensors or biosensors. Examples of data derived are forexample heart rate or heart rate variability, skin conductivity, brainactivity, muscle action, respiration, voice tone, movement pattern,language use, or the like. Data capture devices or techniques are, forexample, an electrocardiogram, electrodermogram, electroencephalogram,electromyogram, thermometer, photoplethysmogram, pneugraph, or the like(see also capture device of FIG. 17). One or more physiological sensedannotation data elements could be automatically derived by the computerdevice, the computer or application server or online service based oncontext sensing or sensing changes in the physiological sensed data.

Once the network of enriched lifestreams is established one coulddetermine relevance of the enriched lifestreams or aspect of theenriched lifestreams by determining density of related connections inthe network, or by determining any data elements that define or are partof the enriched lifestream data. This could for example be contextualinformation (e.g. people, places, business, social network, etc.),user-defined information or goals, or user intentions or emotions asthey are defined in the streams. Relevance could be determined by thecomputer or application server or by the online service. Based on eitherautomatically defined relevance the enriched lifestreams or aspects ofthe enriched lifestreams could be ranked and as such presented to theuser on their computer device.

As one of ordinary skill in the art will appreciate, various changes,substitutions, and alterations could be made or otherwise implementedwithout departing from the principles of the present invention, e.g.notes can be communicated to any service capable of being connected tothe network, and any network, such as a WAN or LAN, can be used inaddition to or in replacement of the Internet. Accordingly, the scope ofthe invention should be determined by the following claims and theirlegal equivalents.

What is claimed is:
 1. A method comprising: receiving, by at least onecomputer processor, a first stream of note data items, a second streamof recording data items, and a third stream of annotation data items;determining, by the at least one computer processor, that: a pluralityof recording data items of the second stream should be linked to a firstplurality of note data items of the first stream, and a plurality ofannotation data items of the third stream should be linked to a secondplurality of note data items of the first stream; linking, by the atleast one computer processor: the plurality of recording data items withthe first plurality of note data items, and the plurality of annotationdata items with the second plurality of note data items to create anetwork of linked data items; and determining, by the at least onecomputer processor, a relevance of a particular data item within thenetwork of linked data items based on a density of related connectionsto the particular data item in the network of linked data items.
 2. Themethod of claim 1, wherein the annotation comprises physiological senseddata of a user.
 3. The method of claim 2, wherein the network of linkeddata items are part of a health tracking service.
 4. The method of claim2, wherein the physiological sensed data includes heart rate datagathered from the user.
 5. The method of claim 1, wherein the annotationcomprises externally sensed data of an environment of a user.
 6. Themethod of claim 5, wherein the externally sensed data indicates movementthrough the environment.
 7. The method of claim 6, wherein the movementdata is determined, at least in part, based on location data gatheredfrom a Global Positioning System (GPS) device.
 8. The method of claim 1,wherein each note data item in the first stream is associated with adistinct time period.
 9. The method of claim 1, further comprising:determining, based on the network of linked data items, that a user hasmet a first predetermined goal for a specified time period; andpresenting an indication that the user has met the first predeterminedgoal.
 10. The method of claim 1, wherein the determining that theplurality of annotation data items should be linked to the secondplurality of note data items is based on commonalities in the pluralityof annotation data items and the second plurality of note data items oris based on preferences.
 11. The method of claim 1, comprising:detecting a change in context by the at least one computer processor;and in response to the detecting the change in context, deriving any ofthe data items.
 12. The method of claim 11, wherein the context includeschanges related people, faces, location, direction, keywords, logicalcontent blocks, identity of person or place said user is looking at,externally sensed data or a combination thereof.
 13. The method of claim1, wherein the first plurality of note data items overlaps with thesecond plurality of note data items.
 14. The method of claim 1, whereineach note data item in the first stream includes text and an indicationof a time associated with generation of the note data item.
 15. Themethod of claim 1, wherein: the first stream of note data items includestext, audio media, visual media, audio-visual media, recorded data, aweblink, a pointer to an information source, or a combination thereof;the second stream of recording data items includes audio datarecordings, video data recordings, audio-video data recordings, or acombination thereof; and the third stream of annotation data itemsincludes data describing an environment of a user, data describingphysiological sensed data of the user, or a combination thereof.
 16. Asystem comprising: at least one computer processor; and at least onememory containing instructions that, when executed, cause the computerprocessor to: receive, by the at least one processor, a first stream ofnote data items, a second stream of recording data items, and a thirdstream of annotation data items; determine, by the at least oneprocessor, that: a plurality of recording data items of the secondstream should be linked to a first plurality of note data items in thefirst stream based on preference settings or commonalities in theplurality of recording data items and the first plurality of note dataitems, and a plurality of annotation data items of the third streamshould be linked to a second plurality of note data items of the firststream based on the preference settings or commonalities in theplurality of annotation data items; linking, by the at least onecomputer processor: the plurality of recording data items with the firstplurality of note data items, and the plurality of annotation data itemswith the second plurality of note data items to create a network oflinked data items; and provide, by the at least one processor, a searchfunction to a user for searching the network of linked data items andfor returning at least one data item from the network of linked dataitems that is relevant to a search query.
 17. The system of claim 16,wherein each data item in the network of linked data items is associatedwith a distinct time period.
 18. The system of claim 16, wherein theinstructions further cause the at least one computer processor to:determine, based on the network of linked data items, that the user hasmet a first predetermined goal for a specified time period; and presentan indication that the user has met the first predetermined goal.
 19. Anon-transitory computer-readable medium including instructions that,when executed by at least one computer processor, cause the at least onecomputer processor to: receive by the at least one processor, a firststream of note data items, a second stream of recording data items, anda third stream of annotation data items; determine, by the at least oneprocessor, that: a plurality of recording data items of the secondstream should be linked to a first plurality of note data items in thefirst stream based on preference settings or commonalities in theplurality of recording data items and the first plurality of note dataitems based on commonalities in the at least two plurality of recordingdata items and the first plurality of note data items or based onpreferences, and a plurality of annotation data items of the thirdstream should be linked to a second plurality of note data items of thefirst stream based on commonalities in the plurality of annotation dataitems and the second plurality of note data items or based onpreferences; create, by the at least one processor, a network of linkeddata items by linking the plurality of recording data items with thefirst plurality of note data items and by linking the plurality ofannotation data items with the second plurality of note data items; andprovide, by the at least one processor, a search function to a user forsearching the network of linked data items and for returning at leastone data item from the network of linked data items that is relevant toa search query.
 20. The non-transitory computer-readable medium of claim19, wherein the third stream of annotation data items includes at leastone data item including physiological sensed data of the user.
 21. Thenon-transitory computer-readable medium of claim 20, wherein the networkof linked data items are part of a health tracking service.
 22. Thenon-transitory computer-readable medium of claim 20, wherein thephysiological sensed data includes heart rate data gathered from theuser.
 23. The non-transitory computer-readable medium of claim 19,wherein the third stream of annotation data items includes at least onedata item including externally sensed data of an environment of theuser.
 24. The non-transitory computer-readable medium of claim 23,wherein the externally sensed data indicates movement through theenvironment.
 25. The non-transitory computer-readable medium of claim24, wherein the movement data is determined, at least in part, based onlocation data gathered from a Global Positioning Service (GPS) device.26. The non-transitory computer-readable medium of claim 19, whereineach data item in the network of linked data items is associated with adistinct time period.
 27. The non-transitory computer-readable medium ofclaim 19, wherein the instructions further cause the computer processorto: determine, by the at least one processor, based on the network oflinked data items, that the user has met a first predetermined goal fora specified time period; and present, by the at least one processor, anindication that the user has met the first predetermined goal.